pkg/steps: pin bundle builds to amd64 nodes - #5329
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughBundle source and bundle-image builds now pin their build pods to AMD64 while preserving existing node selector entries. Tests validate the helper against single- and multi-architecture pipeline build construction. ChangesBundle build architecture
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 16 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (16 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
This PR addresses intermittent src-bundle / bundle image build failures on heterogeneous (amd64 + arm64) build clusters by ensuring single-arch bundle-related OpenShift Builds are scheduled onto amd64 nodes, avoiding “no image found in manifest list for architecture arm64” failures when inputs only contain amd64 images.
Changes:
- Pin
bundleSourceStep.runbuilds to amd64 viakubernetes.io/archnode selector. - Pin bundle image builds in
projectDirectoryImageBuildStep.run(theIsBundleImage()path) to amd64 via the same selector. - Update comments to document the manifest-list / scheduling failure mode that this resolves.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/steps/project_image.go | Pins bundle image builds to amd64 nodes to avoid arm64 scheduling failures when consuming amd64-only manifest entries. |
| pkg/steps/bundle_source.go | Pins src-bundle (bundle-source) builds to amd64 nodes to prevent buildah failures caused by landing on arm64 nodes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kaovilai The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
6c97adc to
f97451d
Compare
f97451d to
416c24f
Compare
Bundle-source and bundle image builds are single-arch by design and are created without a node selector, so on heterogeneous build clusters the build pod can be scheduled onto an arm64 node. The build inputs (e.g. pipeline:src) are manifest lists that only contain an amd64 entry for single-arch repositories, so buildah then fails with: error creating buildah builder: choosing an image from manifest list docker://.../pipeline@sha256:...: no image found in manifest list for architecture "arm64", variant "v8", OS "linux" Pin these builds to amd64 nodes with a kubernetes.io/arch node selector, matching what constructMultiArchBuilds does for pipeline builds. Fixes openshift#5328 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Tiger Kaovilai <passawit.kaovilai@gmail.com>
416c24f to
0d57bf8
Compare
|
Note Responses generated with Claude Retriggering infra flakes (test pods Pending >1h, 0/52 nodes available) and the unrelated /test checkconfig |
|
Scheduling tests matching the |
|
Note Responses generated with Claude CI status: The remaining failures — This is repo-wide, not from this PR: PR #5278's e2e runs today fail identically ( |
|
Note Responses generated with Claude Root cause of the |
|
Note Responses generated with Claude #5331 merged and the fixed ci-operator image promoted — retesting the jobs broken by the credential validation regression: /test breaking-changes |
|
Note Responses generated with Claude Post-#5331 retest results:
— the e2e-oo fixture references the long-EOL |
|
Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage. |
1 similar comment
|
Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage. |
| // builds to one architecture is only safe while every pipeline manifest list is guaranteed to | ||
| // contain that architecture. If any assertion below fails, that guarantee changed — update | ||
| // pinBuildToSingleArchNode together with the change that broke it. | ||
| func TestSingleArchBuildPinMatchesPipelineArchGuarantees(t *testing.T) { |
There was a problem hiding this comment.
It seems that the first 3 subtests don't exercises the new code at all. Is there any reason we want to do this?
There was a problem hiding this comment.
Good catch — first 3 subtests don't call pinBuildToSingleArchNode directly. They guard the pipeline-side invariants the pin depends on (default build arch, ResolveMultiArch(), multi-arch build set), not the pin function itself. Only the 4th subtest (pinning merges into an existing node selector) exercises the pin directly. Added per-subtest comments in 76e8e98 to make that explicit.
Each of the first 3 subtests in TestSingleArchBuildPinMatchesPipelineArchGuarantees asserts a pipeline-side invariant the pin depends on rather than calling pinBuildToSingleArchNode directly. Add a per-subtest comment explaining that, addressing review feedback. Signed-off-by: Tiger Kaovilai <passawit.kaovilai@gmail.com>
TestBundleSourceStepRunPinsBuildToSingleArchNode exercises bundleSourceStep.Run end-to-end and inspects the actual build object submitted through the fake BuildClient, rather than the pipeline-side invariants TestSingleArchBuildPinMatchesPipelineArchGuarantees checks in isolation. A pendingOnCreateBuildClient wrapper stamps newly created builds as Pending so waitForBuild's pending-check fires deterministically instead of hanging forever on a build that never leaves its zero-value phase. Signed-off-by: Tiger Kaovilai <passawit.kaovilai@gmail.com>
|
@kaovilai: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Flake root cause found. Real fix already open: #5323 (mock release API, fixed pullspec). Depend on #5322 (already merged). Retest here no fix this till #5323 land — blind retest only waste CI cycle. Note Responses generated with Claude |
Note
Responses generated with Claude
Summary
Bundle-source (
src-bundle) and bundle image builds are single-arch by design and were created without a node selector, so on heterogeneous build clusters the build pod can land on an arm64 node. The build inputs (e.g.pipeline:src) are manifest lists that contain only an amd64 entry for single-arch repositories, so buildah then fails with:This PR pins both bundle build sites (
bundleSourceStep.runinpkg/steps/bundle_source.goand theIsBundleImage()branch inpkg/steps/project_image.go) to amd64 nodes via a sharedpinBuildToSingleArchNodehelper that merges akubernetes.io/archentry into the build's node selector (merge rather than replace, so a selector set earlier on the build would survive), matching whatconstructMultiArchBuildsalready does for pipeline builds. A tripwire unit test (TestSingleArchBuildPinMatchesPipelineArchGuarantees) locks the pin to the pipeline-side guarantees it depends on.Fixes #5328
Why the previous "no node selector" behavior is wrong
The removed comment said the build "will be scheduled on any available node no matter the architecture" — that was written when it was harmless, but since pipeline outputs became manifest lists (#3587, reworked in #5004), the content buildah pulls is resolved by the architecture of the node the pod happens to land on. For a single-arch repo the manifest list has no arm64 entry (amd64 is the only architecture built unless more are configured), so the build's correctness depends on scheduler placement: amd64 node → works, arm64 node → hard failure. All 5 in-run retry attempts can land on arm64 too (
FetchImageContentFailed×5). That is a flake by construction, not a valid "any node" invariant.Evidence (fail → retry-pass on the same PR/job)
From openshift/oadp-operator#2313:
pull-ci-...oadp-dev-4.22-e2e-test-cli-aws: FAILURE 2026-07-23 00:48 UTC (exact error above in build-log) → retry SUCCESS 02:34 UTCpull-ci-...oadp-dev-4.23-ci-index: FAILURE 2026-07-22 22:40 UTC (same error, thensrc-bundle failed after 5 attempts ... FetchImageContentFailed) → retry SUCCESS 22:58 UTCci-indexpair the next day: FAILURE 13:19 UTC → retry SUCCESS 13:33 UTCFarm-wide: search.ci,
no image found in manifest list for architecture, build-logs, 24h → 93 job-runs in 24 hours, e.g. kubevirt/hyperconverged-cluster-operator#4421, assisted-service periodic, csi-operator periodic, plus openshift-kni/, medik8s/, lvm-operator.Why pinning to amd64 cannot break anyone (including
additional_architecturesrepos)additional_architecturesis strictly additive —pkg/api/types.go: "AdditionalArchitectures is a list of additional architectures to build for. AMD64 architecture is included by default." There is no knob to remove amd64, andprojectDirectoryImageBuildStep.ResolveMultiArch()insertsNodeArchitectureAMD64unconditionally before adding configured capabilities. So a repo withadditional_architectures: [arm64]produces apipeline:srcmanifest list containing both amd64 and arm64 — the amd64-pinned bundle build resolves its amd64 entry and works exactly as for single-arch repos. The failing direction (node arch missing from the list) cannot occur in reverse, because no configuration produces a manifest list without an amd64 entry.srcalready requires amd64 capacity:constructMultiArchBuildspinssrc-amd64(and every other pipeline build) tokubernetes.io/arch: amd64nodes. If a cluster could not schedule the pinned bundle build, it could not have builtsrcin the first place.index_generator.go: "At the moment, we support only amd64"), and bundle image content is architecture-independent metadata/YAML — the node arch never affects the produced bundle, only whether its inputs can be pulled.TestSingleArchBuildPinMatchesPipelineArchGuarantees(inbundle_source_test.go) asserts the invariants above against the real functions: the defaultconstructMultiArchBuildsoutput targets the pinned arch,ResolveMultiArch()contains it even when only other architectures are configured, multi-arch pipelines still produce a build for it, and pinning merges into (not replaces) an existing node selector. If a future change breaks any link in that chain, this test fails and points back atpinBuildToSingleArchNode.Test plan
go build ./pkg/steps/...gofmt -lcleango test ./pkg/steps/...passes, including newTestSingleArchBuildPinMatchesPipelineArchGuarantees/cc @droslean
Summary
src-bundleand bundle image builds run only on amd64 (stable/amd64) nodes, even when the inputs come frompipeline:src/manifest lists.src-bundlefailures where buildah pods scheduled onto arm64 couldn’t resolve amd64-only manifest-list inputs.pinBuildToSingleArchNodehelper that pinscorev1.LabelArchStabletoapi.NodeArchitectureAMD64and merges with any existingNodeSelector:bundleSourceStep.runinpkg/steps/bundle_source.goIsBundleImage()branch inpkg/steps/project_image.goTestSingleArchBuildPinMatchesPipelineArchGuarantees) to ensure the pinned “stable/amd64” invariant remains aligned with the pipeline’s multi-arch manifest-list construction and that selector pinning doesn’t overwrite preexisting selectors.